INTRODUCTION

A fitting test environment is required for dynamic testing of a test object (running software). Setting up and maintaining the test environment represents an expertise of which testers generally have no knowledge. This is why a separate department – outside the project – is generally responsible for setting up and maintaining the test environment. Testers are, however, heavily dependent on the test environment – no test can be executed without a test environment.

This guideline discusses in greater detail what a test environment is and what its setup and maintenance look like. Section Tips - Setting Up Test Environments describes the setup requirements for test environments. It also discusses the factors that determine the setup. The section Problems In Test Environments describes typical problems relating to test environments, followed by two solutions to prevent these problems: the DTAP model and three management processes (Configuration Management, Change Management and Release Management). Section Two Special Test Environments then describes two special types of test environment. The points of concern when the test environment is outsourced are described in section Test Environments When Outsourcing. Set Up And Maintain Test Environment As A Service (SP) explains how a permanent test organisation can set up a service for setting up and maintaining a test environment.

TEST ENVIRONMENTS EXPLAINED

Hardware refers to all the tangible parts of a computer (screen, hard disk, network card, etc.). Test environment software refers to all the programs that should be present on the available hardware in order to run the software under test, such as operating programs, DBMS, network and other support programs. Connections are everything that is required to allow the test object to communicate with other systems. The environment data is the set of data that the test environment requires to be able to work with these (user profiles, network addresses, root tables, etc.). Maintenance tools are tools that are required specifically to keep the test environment operational, and management processes are all the activities that are carried out around the setup and maintenance of a test environment.

The setup and composition of a test environment depend on the aim of the test. The success of a test environment depends on the degree to which it can be determined to what extent the test object meets the requirements. Every test may have a different aim, which is why every test can use a different test environment. A unit test, for instance, requires a completely different configuration of the test environment than a production acceptance test.

Sometimes a test environment has a limited size (e.g. one single PC when testing a small accounting package), while sometimes it involves a huge collection of hardware and software, interfaces and procedures, set up in many different sites (e.g. for testing the reservation system of an airline company). In addition to the test level and test type, other aspects - like the maintenance standards, the type of application, the organisation structure and, not least, the available budgets - play an important part.

Test environments represent a critical success factor for virtually every automation project. There are various reasons for this. For instance, in a production environment the maintenance processes have been established for a long time and are still being improved. This does not apply to a test environment. Processes are not yet or partly established, and this may often vary per department and platform. The complexity increases further if the test environment also uses new technologies that have not yet been taken into production and with which the organisation therefore has less experience.

Another development in recent years is that applications use an increasing number of different types of hardware and software. When setting up a test environment for this type of applications, this is translated to a chain of different hardware and software configurations with mutual interfaces. The metaphor ‘the chain is as strong as its weakest link’ then holds true. If one configuration or interface in the chain fails, the entire chain is useless and complete testing is impossible.

Furthermore, a problem or bottleneck in a test environment is not always quickly solved by an administrator. After all, production always has the priority. This is neglecting the fact that delays in the test process result in delays in commencement of production. Such delays can have the same (or worse) consequences as defects that occur in production.